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EFFECTS  OF  DIFFERED  CENIRDL  MECHANISMS  UPON  USE  OF  A  TRAINING  DEVICE 


EXECUTIVE  SUMMARY 


Requirement: 

A  continuing  challenge  to  the  Army  is  to  provide  effective  training  with 
little  or  no  increase  in  resources.  One  training  innovation  being  explored  by 
Fort  Knox's  Training  Technology  Field  Activity  (TTFA)  is  the  Platoon-level 
Battlefield  Simulation  (PIES)  training  device.  Previous  research  (Graham, 
1987)  indicated  that  this  training  device  could  help  the  Array  to  training 
command,  control,  and  communication  skills.  Researchers  were  concerned,  how¬ 
ever,  about  the  efficiency  of  using  the  system's  responding  mechanisms  for 
driving  vehicles  on  the  PIES  system.  This  research  examined  the  relative 
effects  of  different  joystick  systems  ("Driver"  and  "Pacman")  and  different 
computer  response  times  (2.5  seconds  and  1.5  seconds)  on  soldiers'  abilities 
to  drive  a  vehicle  on  the  PIES  system.  The  Driver  joystick  system  was  de¬ 
signed  so  that  all  driving  movements  were  made  from  the  view  of  a  hypothetical 
Driver  inside  the  vehicle.  The  Pacman  control  system  design  was  similar  to 
that  of  the  Pacman  game. 


Procedure: 

Thirty-four  subjects  participated  in  this  experiment.  These  subjects 
completed  two  different  driving  tasks  across  different  routes  on  the  PIES 
terrain.  One  road  march  went  north-to-south  (top-to-bottcsn  on  the  screen)  and 
the  other  march  went  south-to-north.  The  experimental  sessions  consisted  of 
four  trials  with  two  trials  for  each  task.  The  two  trials  for  the  north-south 
route  were  always  completed  first.  The  subjects  used  a  different  type  of 
responding  mechanism  for  each  trial.  The  order  of  using  a  particular  response 
mechanism  was  counterbalanced  to  control  for  any  possible  trial  effects. 

The  responding  mechanisms  were  as  follows:  (a)  a  Pacman  joystick  con¬ 
figuration  with  a  computer  response  time  of  1.5  seconds;  (b)  a  Pacman  joystick 
configuration  with  a  computer  response  time  of  2.5  seconds;  (c)  a  Driver  joy¬ 
stick  configuration  with  a  computer  response  time  of  1.5  seconds;  (d)  a  Driver 
joystick  configuration  with  a  computer  response  time  of  2.5  seconds.  This 
experiment  was  thus  a  2  X  2  within  subject-design  with  two  levels  of  joystick 
configuration  (Driver  versus  Pacman)  and  two  levels  of  computer  response  time 
(fast  versus  slew) . 


Findings: 


A  multivariate  analysis  of  variance  (MANOVA)  indicated  a  significant 
overall  main  effect  for  the  joystick  configuration  variable.  The  subsequent 
univariate  F-tests  indicated  that  the  subjects7  mean  Problem  Scores,  Off-path 
Scores,  and  Time  Off-path  Scores  ware  significantly  lower  for  the  Pacman 
systems  than  for  the  Driver  systems.  Hie  Pacman  joystick  configurations  led 
to  fewer  driving  problems  and  seemed  to  be  easier  to  use  than  the  Driver 
joystick  configurations.  Also,  the  subjects7  verbal  reports  indicated  that 
Pacman  joy-stick  configurations  were  easier  to  use  than  the  Driver  joystick 
configurations . 


Utilization  of  Findings: 

Based  on  these  findings,  the  PIES  was  equipped  with  a  Pacman-style 
joystick  configuration.  This  experiment's  findings  also  demonstrated  the 
importance  of  the  maneuver  control  system  in  any  combat  simulator.  This 
report  contains  information  that  will  be  useful  to  military  and  civilian 
personnel  engaged  in  developing,  implementing,  and  optimizing  training 
simulators  similar  to  FL8S. 
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EFFECTS  OF  DIFFERE  NT  OONIRDL  MECHANISMS  UPCT  USE  OF  A  TRAINING  DEVICE 


Introduction 

A  major  challenge  to  the  modern  Army  is  to  provide  more  effective 
training  while  using  fewer  resources.  This  challenge  emphasizes  the  need  to 
more  fully  exploit  available  training  technologies.  To  meet  this  need,  the 
Army  Training  and  Doctrine  Command  (TRADGC) ,  the  United  States  Army  Armor 
Center  (USAARMC) ,  and  the  Army  Research  Institute  (ARI)  have  established  a 
joint  Training  Technology  Field  Activity  (TTFA)  at  Fort  Knox,  KY.  The  mission 
of  this  TTFA  is  to  identify,  develop,  and  implement  a  variety  of  new 
technology-based  training  methods.  ARI's  mission  within  TTFA  is  to  conduct 
research  on  the  methods  for  using  this  technology. 

One  such  training  innovation  developed  by  Fort  Knox's  TTFA  was  a  system 
of  computer  networked  simulators — Simulation  in  Combined  Arms  Training 
(SIMCAT) — to  provide  training  in  command,  control  and  communications  (C3) 
skills.  SIMCAT  was  created  because  training  C3  skills  in  a  field  environment 
has  become  too  expensive.  The  cost  of  operating  and  maintaining  a  tank  is 
estimated  to  be  nearly  $230  per  hour  (Kristiansen,  1987) .  Another  reason  for 
developing  these  devices  is  that  the  less  expensive  table  exercises  and  board 
games  have  not  faithfully  reproduced  the  C3  problems  inherent  in  combat 
(Kristiansen) . 

SIMCAT  was  thus  developed  to  duplicate  as  many  field  exercise  C3  require¬ 
ments  as  possible  in  a  low  fidelity,  low  cost,  battlefield  simulation 
(Kristiansen,  Roger,  &  Graham) .  The  SIMCAT  system  involved  the  network¬ 
ing  of  seven  IBM  PC  microcomputers.  Four  of  these  terminals  were  used  by  the 
SIMCAT  students.  These  terminals  were  configured  to  control  tank  vehicles 
with  one  tank  per  student  terminal.  A  terminal  was  used  by  the  controller/ 
instructor  and  another  was  used  by  the  OPPOR.  The  seventh  terminal  was  a 
file  server. 

At  the  student  terminals,  a  simulated  driver  and  gunner  were  used  for 
moving  the  vehicle  and  firing  at  the  enemy.  Control  over  the  driver/gunner 
was  exercised  through  a  voice  recognition  system  or  function  pad.  Responses 
from  the  simulated  crew  were  prerecorded  and  programmed  for  appropriate 
playback.  Research  conducted  by  Graham  (1987) ,  Kristiansen  (1987) ,  and 
Lanpton  (in  preparation)  has  demonstrated  SIMCAT '  s  effectiveness  for  training 
C3  skills. 

However,  same  problems  with  SImCaT  were  discovered  (Kristiansen,  1987) . 
The  most  pressing  problem  was  that  the  students  experienced  same  difficulty 
with  controlling  the  tanks.  These  difficulties  were  due  to  problems  with 
using  a  voice  recognition  system  for  this  task.  It  took  the  students  ap- 
pi.O'dmately  twenty  to  thirty  minutes  to  register  their  voices  with  the  voice 
recognition  system.  Additional  time  was  required  to  train  the  students  to  use 
the  voice  recognition  system  in  order  to  control  their  vehicles.  There  were 
also  occasions  in  wh:  h  the  voice  recognition  system  would  not  recognize  a 
particular  driving  command  because  the  student's  voice  changed  under  the 
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stress  of  making  an  immediate  command.  These  problems  with  the  voice  reccgnj 
tion  system  thus  reduced  the  efficiency  of  SIMCAT  to  train  soldiers  in  C3 
skills. 

A  "touchpad"  system  was  then  developed  for  driving  commands.  This 
touchpad  system  was  a  direct  analog  of  the  voice  recognition  system.  As  shown 
in  Figure  1  (taken  from  O'Brien,  1987) ,  the  tank  commander  gave  driving 
commands  by  touching  a  particular  spot  on  the  touchpad.  However,  this 
touchpad  system  was  also  problematic.  One  problem  was  that  the  students  were 
not  able  to  memorize  the  position  of  a  particular  driving  command  on  the 
touchpad.  Hence,  they  were  continually  looking  at  the  touchpad  while  trying 
to  drive  the  tank.  A  second  problem  was  that  the  students  had  sane  difficulty 
in  making  precise  driving  movements  with  the  touchpad.  A  student,  for 
example,  might  accidentally  hit  the  spot  for  a  hard  right  turn  twice  instead 
of  once.  The  tank  would  then  go  in  a  direction  which  was  not  intended  try  the 
students.  A  third  problem  was  that  the  students  frequently  made  several 
inputs  to  the  touchpad  because  of  the  time  delay  between  making  a  control 
input  and  the  system  output.  With  no  immediate  visual  feedback,  the  students 
did  not  realize  that  their  initial  input  was  being  processed  by  the  computer 
system.  These  problems  with  the  touchpad  system  reduced  the  efficiency  of 
SIMCAT  to  train  C3  skills. 


Figure  1.  The  "Touchpad"  system  (taken  from  O'Brien,  1987) 
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The  Platoon-level  Battlefield  Simulator  (PIBS)  was  then  developed  to 
correct  shortcomings  with  the  SIMCAT  system.  Die  modifications  included 
upgrading  the  hardware  and  software  system  to  meat  the  problems  with  SIMM*. 

A  "joystick"  responding  mechanism  was  developed  to  ease  the  previously 
mentioned  problems  with  driving  a  vehicle.  The  PIBS  system  used  the  joystick 
responding  mechanism  as  a  "polling"  device.  Once  during  an  update  cycle  the 
computer  checks  on  the  location  of  the  joystick  and  then  inputs  the  command 
associated  with  this  location. 

The  joystick  chosen — a  Gravis  MK  VI  Joystick  Controller—  was  designed 
to  met  the  basic  ergonomic  standards  for  this  device  (see  Murrell,  1969; 
Woodson,  1981) .  For  example,  this  joystick  was  the  recommended  length  of  6 
inches.  This  system  was  also  designed  so  that  all  driving  movements  were  made 
vis-a-vis  the  view  of  a  hypothetical  driver  inside  the  vehicle.  Hence,  this 
joystick  will  be  referred  to  as  the  "Driver"  joystick. 

The  Driver  style  joystick  was  a  modified  two-axis  joystick  configura¬ 
tion.  The  vertical  (forward-backward)  axis  was  used  for  starting  (forward) 
and  stepping  (backward)  the  vehicle.  The  horizontal  (90  degree  left  and 
right)  axis  was  used  for  making  sharp  right  turns  (90  degrees  right)  and  sharp 
left  turns  (90  degrees  left) .  Students  could  also  make  gradual  right  and  left 
turns  by  placing  the  joystick  at  either  45  degrees  ric^it  or  left.  See  Figure 
2  (taken  from  Perceptronics,  1988)  for  a  pictorial  description  of  this  system. 
See  the  user's  manual  (Perceptronics,  1988)  for  a  more  detailed  description  of 
this  joystick  and  PIBS. 
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Figure  2.  The  "Analog"  joystick  system  (taken  from  Perceptronics, 
1988) 


The  instructions  for  utilization  of  the  Driver  system  were  as  follows:  To 
move  the  tank  out  in  full  speed,  move  the  joystick  totally  forward.  To  ease 
out,  move  the  joystick  halfway  forward.  To  continue  in  the  same  direction, 
let  go  of  the  joystick.  To  make  a  hard  right  or  left  turn,  move  the  joystick 
ninety  degrees  left  or  right.  To  guide  the  tank  15  degrees  left  or  right, 
move  the  joystick  45  degrees  left  or  right.  To  step  the  tank,  pull  hack  on 
the  joystick.  Use  the  appropriate  red  buttons  to  speed  up  or  slow  down  the 
vehicle  at  increments  of  5  KFH.  These  speed  increments  were  arbitrarily  set 
at  5  KFH  by  the  Fort  Knox's  operator . 

Even  though  this  joystick  seemed  simple  to  use,  students  still  had 
problems  with  using  this  system  to  control  their  vehicles.  Lampton  (in 
preparation)  found  that  tank  commanders  had  difficulty  with  using  the  Driver 
joystick  to  stay  in  a  platoon  formation.  This  problem  made  it  very  difficult 
for  the  student  platoon  leaders  to  complete  their  tasks.  Also,  there  was  same 
indication  that  sane  students  might  not  be  able  to  control  their  vehicles  vis- 
a-vis  the  Driver  joystick  system  because  of  problems  with  perceiving  the 
display  from  the  Driver's  position.  Psychologists  (e.g. ,  Witkin,  Moore, 
Goodenouc^i,  &  Karp,  1967)  have  found  that  same  people  tend  to  have  problems 
perceiving  visual  displays  from  the  perspective  of  another  person,  e.g. ,  the 
hypothetical  Driver  in  the  tank.  These  students  may  then  turn  the  tank  to 
correspond  to  their  perspective  rather  than  to  the.  Driver's  perspective. 

A  new  joystick  configuration  was  therefore  designed.  This  joystick  was 
patterned  after  those  used  in  the  Pacman  games  (See  Figure  3) .  That  is,  the 
joystick  configuration  was  a  multiple-placement  system  with  the  student  being 
able  to  place  the  joystick  at  any  location  around  its  circumference.  Move¬ 
ments  for  the  assorted  tank  icons  corresponded  to  the  joystick's  position  as 
viewed  by  t.ie  student.  Also,  the  students  had  to  continually  hold  the 
joystick  in  the  position  in  which  they  wanted  to  drive.  This  system  thus 
differs  from  the  other  joystick  system  in  che  following  ways: 

1.  The  tank's  movements  correspond  to  the  joystick's  position.  A 
student,  for  example,  moving  the  tank  in  a  southwest  position  would  hold  the 
joystick  in  the  southwest  position. 

2.  The  tank's  movements  thus  correspond  to  the  student's  view  of  the 
screen. 

3.  The  student  must  continually  hold  the  joystick  in  the  desired 
traveling  position. 

4.  The  tank  stops  moving  when  then  student  releases  the  joystick. 

Other  aspects  of  this  system  (e.g. ,  pushing  down  on  the  red  buttons  to 
speed-op  or  slow-down  the  vehicle)  were  the  same  as  for  the  Driver  system. 

The  Pacman  joystick  system  was  expected  to  lead  to  fewer  driving  problems 
than  the  Driver  joystick  system.  This  hypothesis,  however,  was  an  empirical 
question.  Even  though  previous  research  has  compared  the  effects  of  different 
two-axis  joystick  systems  upon  subjects'  performance  on  a  computer  task  (Mehr 
&  Mehr,  1972) ,  comparisons  have  not  been  made  between  two  different  types  of 
joysticks. 
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Figure  3.  Ihe  "Pacman"  joystick  system. 

Another  potential  problem  with  using  PLBS  concerned  the  system's 
response  time.  Because  of  the  networking  process  involved  in  the  PLBS  system, 
it  could  take  three  seconds  for  the  students  to  see  the  results  of  their 
commands  on  the  screen. 
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D.  M.  Kristiansen  (personal  communication,  August  25,  1987)  observed  that 
this  response  delay  could  lead  to  driving  problems.  Furthermore,  Norman 
(1983)  has  suggested  that  two  seconds  delay  seems  appropriate  for  any  interac¬ 
tive  computer  system.  Noiman  also  noted  that  the  quicker  the  computer's 
response  tine  the  more  satisfied  users  are  with  the  system.  For  example,  he 
determined  that  users  are  twice  as  satisfied  with  a  computer  response  time  of 
one  second  than  with  a  two  seconds  response  time.  It  was  thought  that  a 
quicker  response  time  would  make  FIBS  a  more  attractive  system. 

Purpose 

The  purpose  of  this  research  was  to: 

1.  Examine  the  relative  effects  of  the  Driver  and  Pacman  joystick 
systems  upon  students'  abilities  to  control  a  vehicle  on  the  PUBS  system. 

2.  Examine  the  relative  effects  of  different  computer  response  times 
(2.5  seconds  and  1,5  seconds)  upon  students'  abilities  to  drive  a  vehicle  on 
the  PIBS  system. 

3.  Examine  the  students'  preference  for  different  joystick  systems  and 
computer  response  times. 


Method 


Subjects 

The  experimental  sample  consisted  of  34  subjects.  Thirty-two  of  these 
participants  were  soldiers  from  the  U.S.  Army  Armor  Center  at  Fort  Knox. 

These  subjects  ranged  in  grade  from  E-6  to  0-2  with  a  nearly  equal  number  of 
noncommissioned  and  commissioned  officers.  Also,  the  soldiers'  military 
experience  ranged  from  a  few  months  to  eighteen  years  with  a  median  level  of 
four  years  and  a  mode  of  less  than  a  year.  A  cursory  examination  of  the  data 
did  not  indicate  any  effects  associated  with  this  variability  in  the  subjects' 
military  experience. 

Two  subjects  were  civilians  working  with  the  U.S.  Army  Research 
Institute  at  Fort  Knox.  These  two  civilians  were  part  of  a  graduate  student 
training  program  at  the  Army  Research  Institute.  These  two  civilians'  data 
were  similar  to  the  other  participants. 

Apparatus 

The  FLBS  training  device  was  used  in  this  experiment.  PIBS  was  modified 
for  this  experiment  by  the  development  of  four  different  responding  mecha¬ 
nisms.  As  previously  indicated,  these  responding  mechanisms  were: 

1.  Pacman  joystick  configuration  with  a  computer  response  time  of  1.5 
seconds  (Pacman- fast) 

2.  Pacman  joystick  configuration  with  a  computer  response  time  of  2.5 
seconds  (Pacman-slcw) 


6 


3.  Driver  joystick  configuration  with  a  computer  response  time  of  1.5 
seconds  (Driver-fast) 

4.  Driver  joystick  configuration  with  a  computer  response  time  of  2.5 
seconds  (Driver-slew) 

Tasks 

The  driving  tasks  consisted  of  two  different  road  marches  across  dif¬ 
ferent  routes  on  the  PIBS  terrain.  One  march  went  basically  north-to-south 
(tqp-to-bottam  on  the  screen)  and  the  other  inarch  went  basically  south-to- 
north  ( bottom- to-top) .  Both  routes  were  of  approximately  the  same  length  and 
difficulty,  and  also  required  the  subjects  to  make  approximately  the  same 
number  of  sharp  and  gradual  right  and  left  turns.  The  subjects  were  told  to 
maintain  a  spaed  of  30  KHi  for  both  marches.  The  following  scenario  was  used 
for  both  marches: 

The  tanks  for  the  two  subjects  (see  procedure)  had  split  off  from  the 
platoon  and  needed  to  reach  the  platoon  by  following  the  route  on  the 
map.  Each  subject  was  provided  with  a  terrain  map  of  the  PIBS  database. 
The  subjects  were  told  not  to  veer  from  this  route.  No  shortcuts  were 
allowed.  When  the  subjects  reached  the  other  tank  in  their  platoon,  they 
were  to  form  a  wedge  position  with  iiheir  two  tanks  parallel  to  each  other 
and  slic^itly  behind  the  lead  tank.  They  were  then  to  move  out  in 
conjunction  with  this  other  tank  until  they  reached  a  specified  release 
point. 

Practical  constraints  (e.g. ,  the  computer  scoring  system)  prevented  the 
use  of  a  platoon  with  four  tanks.  The  course  of  the  lead  tank  in  the  wedge 
formation  was  set  by  a  computer  program.  This  tank  was  programmed  to  make  a 
series  of  sharp  and  gradual  left  and  right  turns.  Also,  the  lead  tank's  speed 
was  set  at  30  KEH. 

Procedure 

The  experimental  sessions  consisted  of  four  trials  with  two  trials  for 
each  task.  The  two  trials  for  the  north-south  route  were  always  completed 
first.  The  subjects  use!  a  different  type  of  response  mechanism  for  each 
tried.  The  order  of  using  a  particular  response  mechanism  was  counterbalanced 
to  control  for  any  possible  trial  effects.  The  counterbalanced  order  for 
presenting  the  different  responding  systems  is  presented  in  Table  1. 

Table  1 

The  Order  for  Presenting  the  Joystick  Configurations 

Set  A  (n=8) — Pacman-slow,  Driver-slow,  Pacman-fast,  Driver-fast. 

Set  B  (n=8) — Driver-slow,  Pacman-slow,  Driver-fast,  Pacman-fast. 

Set  C  (n=10) — Pacman-fast,  Driver-Fast,  Pacman-slow,  Driver-slow. 

Set  D  (n=8) — Driver-Fast,  Pacman-Fast,  Driver-slow,  Pacman-slcw. 


Two  subjects  simultaneously  completed  a  road  inarch.  They  were  instructed 
to  work  separately  on  the  different  tasks.  Minimal  oral  instructions  were 
also  provided  on  using  the  different  response  mechanisms  and  on  completing  the 
different  road  marches.  Hie  subjects  were  not  provided  with  any  practice  in 
using  the  different  response  mechanisms  before  completing  a  trial.  This  is 
standard  procedure  for  this  type  of  experiment  (Mehr  &  Mehr,  1972) .  Allowing 
the  subjects  to  practice  might  have  eliminated  any  possible  effects  for  the 
different  independent  variables. 

The  subjects  were  also  encouraged  to  "think-aloud"  while  completing  a 
task.  This  "think-aloud"  process  can  provide  valuable  insights  into  the 
reasons  why  subjects  are  having  problems  with  human-factors  design  (Lewis, 

1985) .  After  completing  the  four  trials,  the  subjects  indicated  the  response 
mechanism (s)  which  they  liked  best  and  least.  They  also  were  asked  to  provide 
a  rationale  for  their  choices. 

Trained  observers  also  recorded  any  problems  that  subjects  had  with  a 
response  mechanism  vis-a-vis  the  following  categories: 

1.  making  turns 

2.  maintaining  proper  speed 

3.  maintaining  proper  formation 

4.  maneuvering  the  joystick 

These  observers  also  provided  written  comments  about  the  subjects' 
problems.  And,  the  observers  recorded  the  amount  of  time  that  a  subject  had 
to  wait  for  the  other  subject  to  reach  the  wedge  position. 

Training  Procedures  for  the  Observers 

Three  observers  were  used  in  this  investigation.  The  following  proce¬ 
dure  was  used  to  train  these  observers: 

1.  A  detailed  set  of  instructions  was  given  to  each  observer. 

2.  The  experimenter  discussed  these  instructions  with  each  observer. 

3.  Each  observer  went  Ihrough  a  series  of  mock  trials. 

The  training  session  was  completed  when  the  observers  went  through  a  mock 
trial  without  making  any  mistakes.  The  experimenter  emphasized  the  importance 
of  refraining  from  helping  or  interfering  with  the  subjects'  performance. 

Criterion  Measures 

The  main  set  of  criterion  measures  dealt  with  the  subjects'  performance 
while  completing  the  tasks.  This  set  consisted  of  the  following  measures: 

1.  Time  Scores  (time  in  minutes  to  complete  the  task) . 

2 .  Problem  Scores  (number  of  problems  recorded  by  the  observers) . 
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3.  Off -Path  Scores  (weighted  number  of  times  that  the  subjects  were  off 
the  prescribed  route) . 

4.  Off -Path  Time  Scores  (weighted  time  in  seconds  that  the  subjects  were 
off  the  prescribed  route) . 

Time  Scores  were  partially  determined  by  a  computer  program  which 
recorded  the  time  for  both  tanks  to  complete  the  road  march.  This  time  was 
then  subtracted  from  any  tine  that  the  subject  spent  waiting  for  the  other 
tank  to  reach  the  formation.  The  Off-Path  Scores  and  Off-Path,  Time  Scores 
were  also  partially  determined  by  a  computer  program.  The  computer  program 
provided  a  subject's  Off-Path  and  Off-Path  Time  data  vis-a-vis  the  following 
categories:  (a)  50-60  meters;  (b)  61  to  120  meters;  (c)  121-180  meters;  and 
(d)  more  than  181  meters.  Because  these  categories  represented  different 
degrees  of  error,  these  data  were  then  factored  by  the  following  weights  for 
each  category:  (a)  one  for  the  50-60  meter  category;  (b)  two  for  the  61-120 

meter  category;  (c)  three  for  the  121-180  meter  category;  and  (d)  four  for  the 
more  than  181  meters  category.  A  subject,  for  example,  who  was  off-path  on 
three  occasions  at  more  than  180  meters  received  an  Off-Path  score  of  12 
(three  times  four) . 

The  data  also  included  the  subjects'  preference  and  think-aloud  data. 

The  preference  data  included  the: 

1.  number  of  times  that  a  response  system  was  chosen  as  being  the  most 
favorite, 


2.  number  of  times  that  a  system  was  chosen  as  being  the  least  favorite, 

3.  comments  made  by  the  subjects  regarding  their  choices  for  the  most 
favorite  and  least  favorite  system. 

The  think-aloud  data  were  the  subjects'  comments  while  completing  the 
tasks. 

Design 

This  experiment  was  a  2  X  2  within-sub j ects  design.  As  previously  indi¬ 
cated,  the  factors  of  this  design  were  two  types  of  joystick  configurations 
(Driver  versus  Pacman)  and  two  levels  of  computer  response  time  (fast  versus 
slow) .  Repetition  occurred  across  both  the  joystick  and  computer  response 
time  factors. 

Data  Analyses 

A  2  X  2  MANOVA  (multivariate  analysis  of  variance)  for  repeated  measures 
was  computed  to  analyze  the  data  for  the  four  performance  measures.  SFPSX, 
subprogram  MANOVA  (SPPS  Inc,  1983) ,  was  vised  to  compute  this  MANOVA.  If  any 
overall  significant  effects  were  found  for  the  MANOVA  test  then  the  ap¬ 
propriate  univariate  F-tests  were  computed.  The  alpha-level  for  the  different 
analyses  was  set  at  .05.  The  subjects'  preference  and  think-aloud  data  were 
analyzed  descriptively. 
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Because  two  subjects  cane  late  to  a  session,  they  did  not  complete  their 
last  trial  (Driver-fast) .  The  subjects'  scores  for  this  trial  were  the  means 
for  their  particular  order  set  for  that  trial.  This  procedure  should  have  not 
effected  the  MANOVA  results.  Otherwise,  the  SPPSX  program  would  have  elimina¬ 
ted  these  subjects'  data  for  the  other  performance  measures.  These  subjects' 
preference  data  were  not  included. 

Results  and  Discussion 

Hie  means  and  standard  deviations  for  the  four  performance  measures  per 
independent  variable  are  presented  in  Table  2.  In  Table  3,  the  means  and 
standard  deviations  for  these  measures  across  the  different  responding 
mechanisms  are  presented. 

Table  2 

The  Means  and  Standard  Deviations  for  the  Four  Performance  Measures  par 
Independent  Variable 


Pacman 

Joysticks 

Driver 

Joysticks 

Fast  Response 
Tire 

Slow  Response 
Time 

X 

S.D. 

X 

S.D 

X 

S.D. 

X 

S.D. 

Time 
in  mins 

19.74 

12.89 

20.70 

8.68 

20.23 

10.69 

20.33 

11.24 

Problem 

Scores* 

9.03 

7.41 

14.62 

9.86 

11.31 

8.98 

12.34 

9.31 

Off-path 

Scores* 

7.66 

4.16 

9.81 

5.41 

8.63 

4.68 

8.69 

4.65 

Off-path  818 
Time  Scores* 

675 

1047 

831 

858 

740 

1005 

785 

*significant  difference  between  the  joysticks  at  the  .05  level 

The  MANOVA  analysis  indicated  a  significant  overall  main  effect  for 
the  joystick  configuration  variable  (see  Table  4) .  The  subsequent 
univariate  F-tests  indicated  a  significant  difference  between  the  Driver 
and  Pacman  systems  for  the  subjects'  Problem  Scores,  Off-Path  Scores,  and 
Off-Path  Time  Scores  (see  Table  5) . 

The  subjects'  mean  scores  for  the  Pacman  systems  across  these  three 
measures  were  significantly  lcwer  than  their  scores  for  the  Driver 
systems.  This  effect  occurred  regardless  of  the  particular  responding 
mechanism  because  the  interaction  in  the  MANOVA  analysis  between  joystick 
systems  and  computer  response  was  not  quite  significant.  The  Pacman 
joystick  systems,  regardless  of  their  response  time,  thus  led  to  more 
efficient  driving  than  did  the  Driver  systems. 
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Table  3 


The  Means  and  Standard  Deviations  for  the  Different  Performance  Measures 
Across  Responding  Mechanisms 


Pacman 

-fast 

Pacman-slow 

Driver 

-fast 

Driver-slow 

X 

S.D. 

X  S.D. 

X 

S.D. 

X  S.D. 

Time  19 . 60 

in  mins 

Problem  9 . 47 

Scores 

13.26  19.82  12.70 

8.48  8.58  6.27 

20.87 

13.15 

7.42 

9.23 

20.78  9.73 

16.09  10.38 

Off-path  7.53 
Scores 

4.28 

7.79  9.59 

9.59 

5.06 

10.53  5.82 

Off-path  854 
Time  Scores 

753 

781  594 

862 

737 

1231  809 

Table  4 

Results  of  the  MANOVA  Analysis 

Effect 

Value* 

Exact  F 

Hypoth 

DF 

Error 

DF 

Sig  of  F 

Joystick 

Configuration 

.61 

11.90 

4 

30 

.00 

Computer  Response 
Time 

.11 

.95 

4 

30 

.45 

Interaction  between  .23 

Variables 

2.20 

4 

30 

.09 

^Values  based  upon  the  Pillais  Test 


Correspondingly  then,  the  Pacman  joysticks  were  easier  to  use  than 
the  Driver  systems.  This  point  is  reinforced  by  the  subjects'  think-aloud 
data.  As  shewn  in  Table  6,  the  single  most  frequent  comment  made  by  these 
subjects  concerned  problems  with  manipulating  the  joystick.  And  most  of 
their  comments  about  manipulating  a  joystick  were  in  reference  to  the 
Driver  systems.  Examples  of  these  comments  were: 

"Seems  when  I  make  a  guide  the  machine  does  a  hard  turn  maybe  cause  I 
hold  it  down  too  hard."  (for  the  Driver  system) 

"Turn  the  wrong  way  .  .  .  can't  do  it  .  .  ."  (for  the  Driver  system) 
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Table  5 


Results  of  Univariate  F-Tests  for  the  Joystick  Configuration  Variable  with 
1  and  33  D.  F. 


Dependent 

Variables 

Hypoth 

ss 

Error 

SS 

Hypoth 

MS 

Error 

MS 

F 

Sig  of 

F 

Time  Scores 

39.46 

3547.99 

39.46 

107.51 

.36 

.55 

Problem  Scores 

1061.76 

2112.23 

1061.76 

64.01 

16.58 

.00 

Off-path  Scores 

156.74 

239.76 

156.74 

7.26 

21.57 

.00 

Time  Off-path 

Scores  1781162.47 

7730502.47 

1781162.47 

234257.65 

7.60 

.01 

"I  feel  frustrated  when  joystick  not  responding  according  to  my  com¬ 
mand."  (for  the  Driver  system) 


"I  don't  like  holding  the  controls  constantly."  (for  the  Pacman 
system) 

"Where  are  we  going?  .  .  .  This  is  not  accelerating.  Equipment  must 
have  been  built  by  the  lowest  bidder"  (for  both  systems) 

Table  6 


Results  of  the  Subjects'  Think-aloud  Data 


Main  Problems 

Pacman 

fast 

Pacman 

slow 

Driver 

fast 

Driver 

slow 

Totals 

Manipulating 

8 

7 

11 

15 

41 

Joysticks 

Misunderstanding 

3 

3 

1 

3 

10 

Directions 

Following 

0 

2 

4 

3 

9 

Route 

Reading  Maps 

3 

0 

1 

7 

Totals 

14 

15 

16 

22 

67 

The  observers  also  noted  that  the  subjects  seemed  to  have  problems 
with  manipulating  the  Driver  systems.  As  one  observer  noted: 


Subject  turns  the  tank  all  the  way  around  before  heading 
in  the  right  direction.  He  laughs  at  mistakes  and  shakes 


his  head.  He  seams  to  be  getting  disgusted  with  it 
(the  tank  continually  moving  in  the  wrong  direction) . 

Several  plausible  hypotheses  can  explain  the  subjects'  problems  with 
manipulating  the  Driver  systems.  For  one  thing,  the  students  seamed  to 
have  experienced  same  difficulties  in  making  a  response  from  the  Driver's 
viewpoint.  Several  subjects  complained  in  their  think-aloud  data  about 
being  disoriented  when  making  a  response  to  the  Driver  response  system. 
These  subjects  had  trouble,  for  example,  realizing  that  turning  right  for 
the  Driver  when  going  down  the  screen  meant  placing  the  joystick  to  their 
left.  This  confusion  was  also  noted  in  the  observational  data.  Second, 
the  Pacman  joystick  may  have  allow©!  the  subjects  to  make  more  precise 
turns.  Several  subjects  noted  that  the  Pacman  systems  were  easier  to  use 
because  of  this  reason.  Third,  the  subjects  might  have  been  more  accus¬ 
tomed  to  the  Pacman-style  systems.  After  all,  nearly  all  of  these 
subjects  reported  having  same  previous  experience  with  a  Pacman-style 
joystick. 

The  subjects  might  then  have  been  predisposed  to  use  and  favor  the 
Pacman  configurations.  The  results  of  the  preference  data,  however, 
indicated  that  this  was  not  the  case.  As  shown  in  Table  7,  the  subjects 
were  nearly  equally  divided  in  their  preference  for  the  Pacman-fast  and 
Driver-fast  response  mechanisms.  Furthermore,  their  choices  for  the  least 
liked  system  were  nearly  equally  divided  between  the  Driver  and  Pacman 
joystick  systems  (see  Table  8) .  The  subjects'  preference  data  were  not 
totally  congruent  with  their  performance  data. 

Table  7 


Subjects'  Choices  of  Most  Favorite  Responding  Mechanisms  by  Trials 


Trial  1 

Trial  2 

Trial  3 

Trial  4 

Totals 

Pacman 

fast 

3 

3 

0 

3 

9 

Pacman 

slow 

1 

3 

1 

2 

7 

Driver 

fast 

3 

3 

4 

1 

11 

Driver 

slow 

1 

3 

1 

2 

7 

Totals 

8 

12 

6 

8 

34* 

*Based  on  32  subjects  with  some  subjects  indicating  more  than  one  system. 

The  issue  of  maneuverability  was  clearly  the  predominant  reason  cited 
for  either  liking  or  not  liking  a  system.  As  previously  indicated,  the 
main  reason  for  liking  the  Pacman  systems  was  because  they  were  easier  to 
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use.  Conversely,  the  main  reason  for  not  liking  the  Driver  system  was 
that  they  were  hard  to  control.  These  findings  provide  same  further 
support  for  the  nation  that  the  Pacman  joysticks  were  easier  to  use  than 
the  Driver  joysticks. 

Table  8 

Subjects'  Choices  of  Least  Favorite  Responding  Mechanisms  by  Trials 


Trial  1  Trial  2 _ Trial  3 _ Trial  4 _ Totals 


Pacman 

fast 

5 

1 

2 

1 

9 

Pacman 

slew 

1 

4 

1 

3 

9 

Driver 

fast 

3 

1 

0 

1 

5 

Driver 

slow 

5 

3 

3 

3 

14 

Totals 

14 

9 

6 

8 

37* 

★Based  on  32  subjects  with  some  subjects  indicating  more  than  one 
system. 


There  were  also  other  reasons  for  the  subjects'  preference  data,  As 
indicated,  the  subjects  also  tended  to  like  a  system  because  it  responded 
quickly  to  their  commands.  This  finding  supports  Norman's  (1983)  nation 
that  the  quicker  the  computer's  response  time  the  more  satisfied  users  are 
with  the  system.  Same  subjects  also  claimed  to  like  the  Driver  systems 
because  they  were  realistic  and  challenging  while  the  Pacman  systems  were 
boring  and  not  very  realistic.  The  subjects'  choice  for  not  liking  a 
system  also  seemed  to  be  associated  with  which  system  was  used  during  the 
first  and  second  trial  (see  Table  8) . 

Finally,  the  results  failed  to  indicate  any  effect  for  the  computer's 
response  time  upon  the  subjects'  performance.  Perhaps  then,  a  greater 
difference  in  response  time  than  one  second  is  needed  to  produce  a 
difference  in  the  subjects'  performance.  Or  perhaps,  differences  in 
computer  response  time  is  not  a  variable  which  has  an  effect  upon  stu¬ 
dents'  driving  performance.  S.E.  Graham  (personal  communication,  December 
8,  1988)  has  argued  for  the  latter  hypothesis  because  of  P1BS'  software. 
The  tine  which  varied  across  the  conditions  was  the  time  for  updating  the 
terminal  screen  after  the  computer  received  a  joystick  command.  However, 
the  time  between  the  screen  up-date  and  the  computer's  acceptance  of  a  new 
command  remained  constant.  The  subjects  thus  had  the  same  amount  of 
available  reaction  time  for  both  (and  any)  conditions  of  computer  response 
time. 


Surmmary  and  Recommendations 

In  summary,  this  experiment's  main  finding  was  that  the  Pacraan 
joystick  system  led  to  more  efficient  driving  than  did  the  Driver  system. 
Using  the  Pacman  system  may  then  help  make  P1BS  a  more  efficient  training 
device.  Using  this  joystick  system  could  enable  students  to  experience 
fewer  problems  and  frustrations  while  driving  their  vehicles.  Also,  more 
time  could  be  spent  training  C3  skills  rather  than  irrelevant  skills, 
such  as  responding  to  the  device.  Based  upon  these  findings  the  PLBS 
training  device  was  equipped  with  the  Pacman  joystick  system. 

This  experiment  also  pointed  exit  the  importance  of  the  maneuver 
control  system  for  any  combat  simulator.  As  previously  indicated,  a  good 
control  system  would  enable  military  students  to  spend  less  time  in 
learning  to  use  the  device  and  consequently  they  would  have  more  time  for 
training.  It  is  thus  recommended  that  future  combat  simulators  be 
designed  with  control  mechanisms  which  are  easy  for  the  students  to  use. 

This  research  has  also  provided  further  support  for  using  multiple 
methods  and  measures  for  studying  human-computer  interactions.  The  think- 
aloud  method,  for  example,  obtained  insights  into  the  reasons  for  the 
experimental  findings.  The  convergence  of  agreement  across  criterion 
measures  added  confidence  to  the  conclusion  about  an  effect  for  the  Pacman 
system.  See  DuBois  and  Smith  (in  press)  for  an  another  illustration  of 
using  multiple  methods  and  measures  to  study  human-computer  interactions. 

This  experiment  also  left  many  questions  unresolved.  Further 
research  is  needed  to  examine  the  generalizability  of  this  experiment's 
findings  to  other  human-computer  situations  and  populations.  Further 
research  is  also  needed  to  examine  the  underlying  reasons  for  this 
experiment's  findings.  Another  experiment  is  needed,  for  example,  to  test 
the  hypothesis  that  the  joystick  effect  was,  partially  at  least,  a 
function  of  same  students'  inability  to  respond  vis-a-vis  the  Driver's 
view.  Ibis  report's  experiment  should  thus  be  viewed  an  exploratory 
investigation  on  important  but  neglected  issues  in  human-computer  interac¬ 
tions. 

Hie  following  recommendations  can  be  drawn  from  this  report: 

1.  PUBS  should  be  equipped  with  a  Pacman  style  joystick  system. 

2.  Future  combat  simulators  should  be  designed  with  control 
mechanisms  which  are  easy  for  the  students  to  use. 

3.  Studies  on  human-computer  interactions  should  use  multiple 
methods  and  measures. 

4.  Further  research  is  needed  to  examine  the  generalizability  of 
this  experiment's  findings  to  other  human-computer  situations  and  popula¬ 
tions. 


5.  Further  research  is  also  needed  to  examine  the  underlying  reasons 
for  this  experiment's  findings. 
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